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ABSTRACT 



This paper offers recommendations to the National Center for 
Education Statistics (NCES) on the development of the background 
questionnaire for the National Assessment of Adult Literacy (NAAL) . The 
recommendations are from the viewpoint of a researcher interested in applying 
sophisticated statistical models to address important issues in adult 
literacy. The paper focuses on five issues, each of which is the subject of a 
section of the paper: sampling; selection bias; measurement; policy modeling; 
and gauging cohort effects. Each section considers the scope of the issue and 
then makes recommendations to NCES. These recommendations include providing 
all appropriate sampling weights in NAAL data; examining contextual effects 
on the distribution of literacy ability in the population; considering 
relevant auxiliary variables that would constitute the selection equation; 
considering the hypothesized number of factors and including at least four 
variables measuring each factor in the questionnaire; obtaining retrospective 
data on general and job-specific literacy-related activities; and exploring 
the possibility of linking NAAL with existing longitudinal surveys. (Contains 
21 references.) (YLB) 
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Abstract 



With the advent of simple-to-use advanced statistical software packages, it is becoming increasingly easy to 
specify and estimate complex statistical models for addressing substantive questions in adult literacy and other areas 
of education. Such problems as the role of literacy in voting behavior or the job market experiences of individuals 
with GEDs, can be addressed with relative ease. However, a thoughtful application of statistical models to 
educational data leads to the recognition that certain assumptions must be met for the model estimates to be useful 
for theoretical explanation and/or policy analysis. The purpose of this paper is to offer recommendations to the 
National Center for Education Statistics on the development of the background questionnaire for the National 
Assessment of Adult Literacy. The recommendations presented in this paper are from the viewpoint of a researcher 
interested in applying sophisticated statistical models to address important issues in adult literacy. This paper will 
focus on five issues: (1) sampling, (2) selection bias, (3) measurement, (4) policy analysis, and (5) cohort effects. 
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Introduction 



With the advent of simple-to-use advanced statistical software packages, it is becoming increasingly easy to 
specify and estimate complex statistical models for addressing substantive questions in adult literacy as well as other 
areas of education. However, a thoughtful application of statistical models to educational data leads to the 
recognition that certain assumptions must be met for the model estimates to be useful for theoretical explanation 
and/or policy analysis. These assumptions concern the sampling of respondents, measurement of constructs, and 
selection bias, to name a few. 

Pursuant to the Call for Papers on the National Assessment of Adult Literacy (NAAL), this paper provides 
technical recommendations to the National Center for Education Statistics (NCES) as it plans for and executes the 
NAAL. These recommendations specifically take into account the application of advanced statistical modeling of 
secondary public use data. 

The technical suggestions provided in this paper are guided by many of the recommendations summarized 
in NCES Working Paper No. 98-17 entitled “Developing the National Assessment of Adult Literacy: 

J Recommendations from Stakeholders’' (hereafter referred to as NCES 98-17). Specifically, this paper emphasizes 
the utility of the background questionnaire for basic research and policy studies. This report addresses the following 
five issues: (1) sampling, (2) selection bias, (3) measurement, (4) policy modeling, and (5) cohort effects. 

1. Issues of Respondent Sampling 



The issue of sampling discussed in this section concerns the ability to draw accurate inferences about the 
population of adults. Fortunately, solutions to this problem are relatively straightforward. Specifically, many national 
educational surveys employ multistage clustered sampling with schools sampled first, followed by the sampling of 
students and teachers within those schools. Drawing proper inferences from data generated from such sampling 
designs not only requires proper statistical modeling, such as multilevel modeling, but also requires the incorporation 
of sampling weights. 
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The issue of sampling weights also applies to data drawn from other multistage cluster designs, such as the 
household survey sample that will be utilized forNAAL. As noted inNCES 98-17, there is a desire to assess 
underrepresented groups based on specific policy needs. These groups include rural adults, adults in welfare -to -work 
programs, Native Americans, blacks, and Hispanics, to name a few. The issue of oversampling is subtle, but clearly 
important if one wishes to make accurate inferences about underrepresented populations. 

In the context of advanced statistical modeling, it is essential that oversampled groups not exert undue 
influence on statistical estimates or conclusions of model adequacy. To mitigate this problem it will be essential to 
provide the sampling weights for all respondents in the survey. Many statistical software packages allow for the 
incorporation of sampling weights in regression-based methodologies. 

1. 1 Issues in Clustered Sampling and Multilevel Modeling 

With the use of multilevel modeling for the analysis of hierarchical data (Bryk and Raudenbush 1992), great 
flexibility now exists to study educational issues in the context of the organizational structure of education. Surveys 
sponsored by the NCES, such as the NELS:88, sample respondents in ways that reflect the natural structure of 
educational organizations, viz., students nested in schools. Indeed, relative to the early discussion, such datasets also 
contain appropriate weights at each level and utilize software programs such as hierarchical linear modeling (HLM) 
(Bryk 1996). 

With regard to the structure of the NAAL background questionnaire, the question is whether the proposed 
sampling design yields natural organizational structures that might lend themselves to the application of multilevel 
modeling. For example, the National Adult Literacy Survey (NALS) defined a four-stage stratified sampling design 
in which primary sampling units (PSUs) consisted of geographic clusters of one or more adjacent counties (first 
stage). This was followed by Census blocks within counties (second stage), households within Census blocks (third 
stage), and finally, adults within households (fourth stage). To the extent that these PSUs represent groups of 
substantive importance to issues of literacy, it may be useful to examine variation in respondent literacy as a function 
of demographic characteristics, such as poverty levels, captured by the PSUs. 
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An application of multilevel modeling to the NALS data was recently reported by Sheehan-Holt and Smith 
(1999). In their analysis, they used census blocks (segments) as the contextual, or level-two, unit of analysis. They 
argued that census blocks, as defined in the NALS sampling design, could serve as a proxy for neighborhood, but 
noted that census block level variables were not collected. In their case, they created a “neighborhood average 
income” based on the income reported by respondents within the segments. 

1.2 Recommendations to NCES 

Recommendations for NCES in the construction of the NAAL survey are twofold. First, it is crucial that the 
public-use NAAL data contain all appropriate sampling weights in order for accurate inferences based on advanced 
secondary statistical analyses to be made. The literature is clear regarding the problem associated with inaccurate 
inferences when sampling weights are ignored. Every effort should be made to encourage the use of sampling 
weights in secondary analyses. 

Second, there is no doubt as to the importance of examining contextual effects on the distribution of literacy 
ability in the population. In terms of the design of the next background questionnaire, every attempt should be made 
to link the NAAL survey to census-based surveys so that more detailed neighborhood variables can be collected. 
Indeed, recent research by Archbald, Kaplan, and Nakib (conducted under OERI grant # R308F60010, 1 996-1998) 
showed how it was possible to link the National Assessment of Educational Progress (NAEP) to census data and 
ultimately to the district data codebook. This allowed for the incorporation of data at the level of the school district to be 
merged with data from the NAEP background questionnaire for the purposes of multilevel modeling. We strongly 
recommend that the NAAL provide similar links to allow for the development of contextual models of adult literacy. 

If interest centers on the variation of respondent literacy as a function of group-level characteristics, such as 

✓ 

GED programs, then it is required that a different form of sampling be employed. For example, some form of 
clustered sampling of programs followed by respondents within programs would allow application of multilevel 
modeling methods. If such a sub-study is of relevance to the larger goals of NAAL, then the questionnaire should 
capture as many features of the group-level variables as possible. With regard to sampling weights, these should be 
made available at both the group and respondent levels. 




12 



3 



One practical recommendation that comes from a consideration of sampling weights is that very clear 
guidelines regarding their use should be provided. That is, NCES should make available in their NAAL 
documentation clear instructions on how to incorporate sampling weights in secondary analyses of the data. 

2. Issues of Selection Bias 

The issue of selection bias refers to problems of inference that arise from the nonrandom participation of 
respondents and their assignment to groups. A recommendation that appeared in NCES 98-17 was that background 
data should provide information regarding participation in adult education programs as well as participation in 
citizenship activities. However, in both cases, individuals are not randomly assigned to groups that either participate 
or do not participate in these activities. As an example, consider the problem of literacy and voting (Kaplan and 
Venezky 1994; Venezky and Kaplan 1998). Before being able to study the impact of literacy on voting behavior, it is 
first necessary to recognize that the lack of random assignment into voting and non-voting groups hinder attempts to 
make claims about the role of literacy on voting behavior. 

The difficulty with asserting claims in this case arises from the fact that those who state that they have voted 
are not a random sample of the general population who are eligible to vote. Thus, we need a model for estimating the 
probability of observing an affirmative response to the voting question. After accounting for age and citizenship 
requirements, it is clear that one cannot vote unless one is registered to vote. However, the probability of registering 
to vote is dependent on many factors that are not necessarily related to voting. Thus, without a voter registration 
question (and relevant predictors of registration) in the background questionnaire, it would be virtually impossible to 
account for nonrandom selection mechanisms within a statistical model of voting behavior. Although the Young 
Adult Literacy Survey (Kirsch and Jungeblut 1986) included a question on voter registration, the previous NALS 
background questionnaire did not include such an item. 

The example of literacy and voting behavior is only one of a number of possible examples where selection 
bias can occur. As another example, consider a comparison of those who obtain General Educational Development 
(GED) credentials with high school graduates. In this case, while it may be reasonable to assume that high school 
graduates are a relatively representative group, it is clear that those who enroll in the GED program and obtain the 
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GED are not. That is, of those who do not obtain their high school diplomas, a nonrandom sample of those 
respondents will enter a course leading to the GED and still another nonrandom sub-sample of that group will earn 
GED credentials. Thus, a model that predicts the probability of obtaining a GED is needed to account for nonrandom 
assignment to these groups. 

2 . 1 Methods for Modeling Selection 

For completeness, it is useful to consider the variety of statistical approaches available to secondary users 
who wish to account for selection bias. Perhaps the most popular approach to the problem of nonrandom selection 
into treatment groups is the analysis-of-covariance (ANCOVA). With ANCOVA, the investigator is required to 
choose one or more concomitant variables (covariates) to be used in the analysis. In the GED example given above, 
one possible covariate might be a measure of the family socioeconomic status of the respondent — the argument being 
that only certain individuals with somewhat higher levels of family SES can afford to return to school to study for the 
GED. Scores on, say, document literacy, are adjusted for their relationship to the covariate(s) and the analysis-of- 
variance is conducted on the adjusted scores (Kirk 1995). 

Although ANCOVA represents a classic approach to the problem of selection bias, it does have certain 
limitations. The most important limitation concerns the ANCOVA assumption of homogeneity of regression. 
Homogeneity of regression refers to the requirement that the relationship between the outcome and the covariate be 
the same for all groups under study. This is difficult from a practical standpoint since, as the number of covariates 
increases, it becomes increasingly unlikely that this assumption will be satisfied. 

Another approach to selection modeling that has its roots in econometrics is the two-step approach of James 
Heckman (1976). The conceptual idea behind two-step modeling is that a substantive equation of interest is 
misspecified if it is missing a variable that accounts for the probability of observing the data. In the voting case, one 
may be interested only in the relationship between document literacy and educational level for those who voted. A 
regression model to study this question would be misspecified because the process of selection creates a disturbance 
term with a mean that is no longer zero and which is correlated with the predictor. 
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To correct for the specification error, two regression equations are specified: a selection model equation and 
a substantive model equation. In the selection model equation, a probit regression is conducted, regressing a 
qualitative group membership variable (voted/did not vote) on a set of variables assumed to predict group 
membership. A new variable is then formed for each person. Referred to as the hazard rate (Heckman 1976) 1 , this 
variable represents the likelihood that an individual will be excluded from the sample. Next, the hazard rate is added 
to the substantive equation of interest and a standard ordinary least squares regression analysis is conducted. 

Although Heckman’s approach is a classic methodology for dealing with selection bias, it is known to suffer 
from statistical problems arising from the incorporation of the nonlinear based hazard rate into a linear equation. An 
approach that derives the conceptual benefit of the Heckman approach without the computational problems, and that 
is similar in many respects to ANCOVA, is the propensity score approach (Rosenbaum 1995; Rosenbaum and Rubin 
1983). 

The propensity score was proposed by Rosenbaum and Rubin as a means for balancing treatment and 
control groups with respect to covariates in nonrandomized experimental studies. The propensity score is based on 
the conditional probability of assignment to a treatment group given a set of observed covariates. 

In a typical application of this approach, each observation is associated with a propensity to be assigned to 
the treatment group. The distribution of propensity scores is then usually divided into strata at the quintile points of 
the distribution and analyses of treatment group differences are conducted within strata 2 . Comparisons of treatment 
group differences within and across strata provide evidence for whether or not the bias due to nonrandom selection 
into treatment groups has been accounted for by the propensity score adjustment. If, for example, groups are found to 
differ by a constant amount across quintiles, this can be taken as evidence that the groups differ beyond what can be 
explained by the process that assigned individuals to those groups. If, on the other hand, groups do not differ across 
quintiles, this can be taken as evidence that the selection mechanism is accounting for the group differences. Finally, 
if the size of the differences between groups varies across strata, this can then be taken as evidence that selection 

1 The hazard rate is derived from the expression of the conditional expectation function under truncation (Goldberger 
1981). This expectation can be written as E(y|x) = x’(3 - gA(z). In this expression z = (c - x’P)/ ca, where c is a 
truncation point, x is a vector of predictors, p is a vector of regression coefficients, and co is the standard deviation of 
the disturbances. The hazard rate X(z) is the ratio of the probability density of z to the cumulative distribution of z 
(Berk and Ray 1982). 

2 For continuous distributions, strata sub-classification at the quintile points has been found to remove at least 90 
percent of the bias due to nonrandom selection effects. 
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characteristics are interacting with group differences yielding differential effects on the outcome of interest. This is a 
classic example of the selection by treatment interaction. 

A review of the substantive literature suggests that the propensity score approach has been used in such 
diverse fields as sociology, medicine, psychiatry, and economics. Recently, Hoffer (1994) used the propensity score 
approach in an analysis of educational tracking and Kaplan (in press) recently extended the propensity score 
approach to latent variable models. 

2.2 Recommendations to NCES 

Considering the ubiquity of selection bias in social and behavioral science research, it is essential that care 
be taken to consider relevant auxiliary variables that would constitute the selection equation. The examples of 
participation in adult education or citizenship activities are prototypical, but it would be relatively straightforward to 
generate other more subtle examples. The ANCOVA, Heckman two-step, and propensity score approaches for 
addressing selection bias have one thing in common — they require the measurement of concomitant variables. Thus, 
as consideration is given to the development of background items for NAAL, it is recommended that, for each 
question that categorizes respondents into groups, thought be given as to how respondents were allocated to those 
groups. Furthermore, auxiliary measures that predict that allocation should be included. 

3. Issues of Measurement 

It is often the case that behaviors and attitudes of respondents are desired. For example, in the NALS, 
questions were posed regarding individuals’ self-perceptions of literacy ability. It may be of interest to determine the 
number of factors underlying self-perception of literacy ability. Such underlying factors, if they exist, will be fewer 
than the original set of variables used to measure those factors, and could be used in more parsimonious models 
linking self-perception to actual literacy skills as measured by the literacy assessments. The development and 
assessment of underlying constructs representing behaviors and attitudes constitutes an important part of secondary 
statistical analysis. 
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If the goal is to validate a set of underlying behaviors and attitudes, then it is essential that multiple 
measures, or observed variables, of those attitudes or opinions be obtained. Generally, the methodology used to 
validate the existence of underlying dimensions is factor analysis. From the point of view of utilizing factor analysis, 
a number of statistical issues need to be raised to help inform thinking about the development of the NAAL 
background questionnaire. 

First, it is necessary to consider the issue of the identification of an underlying factor. The concern here is 
the extent to which a hypothesized factor could, in principle, be rejected on the basis of the observed data. Factor 
analysis requires that, for any given factor, at least three measured variables load on that factor. With respect to 
degrees-of- freedom, such a “three-variables-one-factor” model is just-identified — meaning that the factor loadings 
can be estimated, but the hypothesis of a single underlying factor cannot be formally tested. For the purpose of 
rejecting the hypothesis of a single underlying factor, at least four measured variables are required. In multiple-factor 
models, the situation is a bit less constrained, and it is possible to reject, say, a two-factor model as long as there are 
at least two measured variables per factor and the factors are allowed to correlate. In general, the principle is that in 
the development of a set of items that are hypothesized to measure an underlying set of factors, the more observed, 
or measured variables per factor that can be developed, the better. 

The second issue concerns the metric of the measures. Typical metrics include five-point or seven-point 
Likert scales. The decision regarding the number of scale steps should be made on the basis of substantive 
considerations and a knowledge of the sample of respondents 3 . From a statistical point of view, a general principle is 
that the more scale steps comprising a measure, the better. However, it is no longer entirely necessary that items be 
measured on five-point or seven-point scales. Indeed, developments in factor analysis over the past 15 years allow 
for factor structures to be tested on measures that are dichotomous, ordered categorical (e.g., Likert scales), 
continuous, as well all combinations of these types (Muthen 1978, 1984). 

The statistical requirement pertaining to the scale steps is that there be a hypothesized underlying response 
propensity for each measure. For example, consider the case where one wishes to assess reading habits. The NAAL 
questionnaire could ask respondents whether they read newspapers at least once a week, with the response choice 
being “yes” or “no”. To incorporate this and other similar items into factor analysis, it is assumed that underlying the 
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dichotomous response is a propensity to respond “y es -” The actual observation of a “yes” response occurs after a 
threshold is exceeded. 

When considering the factor analysis of a correlation matrix based on, say, dichotomous response variables, 
it is not correct to simply analyze the Pearson product-moment correlation matrix of the data. Indeed, such an 
analysis could result in the extraction of so-called “difficulty factors” 4 . Instead, it is necessary to calculate different 
types of correlations that account for the scale type under the assumption of the underlying response propensity. For 
example, the correlation between two dichotomous variables with an underlying normally distributed response 
propensity is referred to as a tetrachoric correlation. Similarly, the correlation between two Likert-scale variables 
assuming a continuous response propensity underlying each variable is referred as a polychoric correlation. Under 
the assumption of the underlying response propensity, the analysis of these (and other similar) types of correlations 
provide accurate assessment of the underlying factor structure of the data. 

In the context of confirmatory factor analysis or structural equation modeling, more complicated estimation 
procedures are necessary for the analysis of these types of correlation matrices. A discussion of these estimation 
procedures is beyond the scope of this paper. It is sufficient to say that, without these specific estimation methods, 
which require very large sample sizes, the results may be quite inaccurate. 

3.1 Recommendations to NCES 

From the standpoint of a literacy researcher using NAAL secondary data, the issue of scale type is less 
important than the number of items needed to measure the factor. This is particularly true given the large sample size 
proposed for NAAL. Therefore, when building scales to measure underlying attitudes or behaviors, consideration 



3 For example, in developing opinion or attitude items for young children, dichotomously scored items might be 
chosen because they would be easier for the child to understand. 

4 The problem of “difficulty factors” is a classic psychometric problem arising from the factor analysis of 
dichotomous variables. The Pearson correlation between two such items is referred as the phi-coefficient. The 
problem is that if the dichotomous variables exhibit unequal response frequencies, a factor analysis of such data will 
yield a factor that is an artifact of the unequal responses. 
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should be given to the hypothesized number of factors and that at least four variables measuring each factor be 
included in the questionnaire. 



4. Issues Pertaining to Policy Modeling 

In this section, consideration in given to the use of NAAL data for policy studies. It is clear from a perusal 
of NCES 98-17 that a major purpose of NAAL is to inform federal and state policy on issues of adult literacy. 
Generally, policy-relevant information is conveyed through the use of accurate descriptive statistics and cross- 
tabulations as well as linkages to other relevant data for the purpose of measuring trends in literacy. A discussion of 
linking the NAAL to other existing databases was discussed earlier in the context of respondent sampling. 

Another approach to providing policy-relevant information is the use of advanced statistical models to 
simulate changes in policy-relevant variables and observe the effects of those changes on literacy outcomes of 
interest. I refer to such an exercise as policy simulation modeling , which can be defined as a method by which a 
statistical model is used for prediction purposes that have policy relevance. The use of advanced statistical models 
for policy studies has had a long tradition in economic modeling, but has been lacking in educational research. 
Generally speaking, any statistical model can be used for the purpose of policy simulation modeling. However, the 
conventional approach to statistical modeling in education has been to develop a model, estimate its parameters, test 
the statistical significance of components of the model, and then interpret it. In the case of structural equation 
modeling applied to educational data, it is often found that a process of model modification occurs in an effort to 
bring models that do not fit in line with the data. Rarely, if ever, is the final form of these models used to simulate 
alternative policy or clinical scenarios. Likewise, rarely are models judged on the basis of whether the results gleaned 
from alternative policy scenarios make any substantive sense. 

Recently, Kaplan and Elliott (1997) argued for such an approach and demonstrated policy simulation 
modeling in the context of validating education indicators. In that analysis, Kaplan and Elliott determined a number 
of important issues that are relevant to the development of the NAAL questionnaire. These issues concern obtaining 
predictor, or exogenous, policy-relevant variables and the metric of policy-relevant variables, both of which will be 
discussed, in turn, below. 
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4.1 Obtaining “Exogenous” Policy-Relevant Variables 



In consideration of the use of statistical models for policy simulation modeling, it is helpful to distinguish 
between the entire set of all possible predictor, or exogenous, variables and a subset of those predictor variables that 
may be viewed as policy-relevant. To a certain extent, this discussion is tempered by the fact that NAAL will be a 
cross sectional, as opposed to a longitudinal, study. In any case, it is clear that within the set of exogenous variables, 
certain variables (e.g., age) cannot, in principle, be manipulated by the investigator, while others (e.g., number of 
hours spent in literacy related activities), in principle, can. 

A subtle feature of the problem is the extent to which variables that are considered policy-relevant are truly 
predictive. To a certain extent, the question of the exogeneity of a variable is an empirical one (Richard 1982). Not 
all variables that are termed exogenous truly are, and there are advanced exogeneity tests that allow this assumption 
to be assessed. However, in policy simulation modeling it is important that thought be given to the selection of 
variables that have policy relevance. This selection should only be guided by theory and the interaction of literacy 
policy specialists with survey research specialists and psychometricians. 

4.2 Natural Metrics of Observed Variables 

Another concern that emerges from a consideration of policy simulation modeling is the metric of the 
policy-relevant variables. If secondary statistical modeling is to be used for developing policy-relevant prediction 
models, then it is essential that observed variables be measured in their natural metrics. To be specific, many national 
surveys code sensitive variables into categories that render them almost useless for policy simulation modeling. For 
example, an important variable in an analysis of literacy outcomes is income. Understandably, income is a sensitive 
question. However, in collapsing income into large and arbitrary categories we render this variable useless for policy 
modeling. 

It is understood that such advice may conflict with confidentiality issues. Nevertheless, it is important that 
this issue be discussed in designing the background questionnaire and that the statistical ramifications be understood. 
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Two possible approaches for addressing this issue would be to 1) develop more refined codes and/or 2) create 
restricted use datasets. 

4.3 Recommendations to NCES 

The issue discussed in this section concerns using secondary public-use data for basic policy research. If 
such an endeavor is deemed valuable, then it is essential that policy analysts and researchers supplement descriptive 
statistics with theoretically guided models that have policy relevance. The advanced statistical models are ripe for 
this use but require serious consideration of the types of variables and their metrics to be included in the background 
questionnaire. Thus, it is advised that, in the development of the background questionnaire, policy analysts in the 
field of adult literacy should meet with survey and measurement specialists to consider the development of policy- 
relevant variables for use in policy simulation modeling. 

5. Issues Relating to Gauging Cohort Effects 



Finally, it is important to develop measures that assess the extent of cohort effects. As justification for this, 
consider that a simple analysis of NALS data shows that there is a precipitous decline in literacy skills after the age 
of forty-five. This could very well be due to a differential growth rate in literacy for different age cohorts. Yet, with 
cross-sectional data, this differential growth rate hypothesis is hard to test. The section to follow will consider two 
approaches to aid in assessing cohort effects: obtaining retrospective background questions and linking NAAL with 
existing longitudinal surveys. 

5 . 1 Obtaining Retrospective Background Questions 

Considering that NAAL will be based on a cross-sectional design, the first approach which may be the most 
feasible would be to include retrospective questions in the background questionnaire. Such questions should be 
directed at ascertaining literacy related activities as well as job histories at key points in the lifecycle of respondents. 
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For example, it may be of interest to probe the job histories of respondents and to ascertain the literacy demands 
required for each job. 

Retrospective questions have a number of serious problems that must be kept in mind (Bijleveld and van 
der Kamp 1998). First, it should be noted that the very nature of the response to a retrospective question is 
nonrandom. That is, what is being observed, or measured, are the responses of individuals who can reflect back on 
their histories. Even if the design had been longitudinal, attrition for any number of reasons would render the 
resulting sample of respondents a nonrandom sample. 

Additional problems with retrospective designs relate to the unreliability of the responses. Such unreliability 
stem from the following sources: (1) memory loss, in which respondents simply forget specific events in their life 
histories. To mitigate against this problem, some have advocated the use of checklists to aid in recalling possible 
events; (1) telescoping, wherein respondents tend to report events as having happened more recently than they 
actually happened; and (3) modification to fit a coherent scheme , wherein respondents tend to interpret events to fit 
current perceptions of themselves (Bijleveld and van der Kamp 1998). 

These problems render retrospective reports of questionable validity. However, the problems are mitigated 
by asking respondents to describe factual events to the extent possible. That is, assessments of job histories and 
literacy activities are more reliable than retrospective assessments of feelings and attitudes. So, instead of asking 
someone how he felt about reading when he was 10 years old, ask instead, “What was your first job?” “Did the job 
require reading?” “What types of material were you required to read on your first job?”, etc. Although such detailed 
probing questions will not eliminate these biases, the strategy is known to mitigate them. 

5.2 Linking NAAL With Existing Longitudinal Surveys 

In the absence of conducting a longitudinal survey, a second approach to gauging cohort effects would be to 
consider linking a sub-sample of NAAL respondents to an ongoing longitudinal survey, such as the NCES National 
Educational Longitudinal Survey (NELS), the NCES High School and Beyond (HSB), or the U. S. Department of 
Labor’s National Longitudinal Survey (NLS). To be certain, the feasibility of such a proposal is unclear. In addition, 
the literacy assessment contained in NAAL is not available in other surveys. Yet, with the development of 




13 



22 



longitudinal statistical methodologies such as growth curve modeling (Muthen and Curran 1997) it is possible to 
examine growth in one domain (e.g., reading achievement) as it pertains to the prediction of a distant outcome such 
as document literacy. Moreover, linkages to other surveys would not require excessively large sample sizes. The 
benefits of seriously considering such linkages is that surveys such as NELS or NLS contain information regarding 
reading practices and other schooling measures. In addition, surveys such as NELS contain background parent 
questionnaire data that might be useful in examining predictors of adult literacy. 

5. 3 Recommendations to NCES 

To gauge the extent of cohort effects, it is suggested that the NAAL background questionnaire obtain 
retrospective data on general and job-specific literacy related activities. It is advisable to construct retrospective 
questions that will prompt factual information to the fullest extent possible. Moreover, it is advisable to include 
possible checklists to aid in the recall of historical events. In addition, NCES should explore the possibility of linking 
a sub-sample of the NAAL respondents to existing surveys sponsored by NCES or the U.S. Department of Labor 
(e.g., NELS, HSB, NLS). Finally, NCES should explore the possibility of obtaining a sub-sample of each age cohort 
of the NALS survey and include these in the NAAL survey. 
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96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1 995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

1 999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education 
Statistics 

Adult literacy — see Literacy of adults 
American Indian - education 

1 999-13 1 993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 

Indian Affairs (BIA) Restricted-Use Codebook 

Assessment/achievement 

95-12 Rural Education Data User’s Guide 

95-13 Assessing Students with Disabilities and Limited English Proficiency 

97- 29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97- 44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

Civic participation 

97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94- 05 Cost-of-Education Differentials Across the States 

Course-taking 

95- 12 Rural Education Data User’s Guide 
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NCES contact 



98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 
1 999-05 Procedures Guide for Transcript Studies 

1 999-06 1 998 Revision of the Secondary School Taxonomy 

Crime 

97- 09 Status of Data on Crime and Violence in Schools: Final Report 

Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 

Customer service 

1 999- 10 What Users Say About Schools and Staffing Survey Publications 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1 999 AAPOR Meetings 



Data quality 

97-13 Improving Data Quality in NCES: Database-to-Report Process 

Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1 999 AAPOR Meetings 



Design effects 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 

Dropout rates, high school 

95- 07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and 

NELS:88 Sophomore Cohort Dropouts 

Early childhood education 

96- 20 1 991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1 995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1 999- 0 1 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

Employment 

96-03 National Education Longitudinal Study of 1 988 (NELS:88) Research Framework and 
Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 
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No. 
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Faculty - higher education 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Finance - elementary and secondary schools 

94- 05 Cost-of-Education Differentials Across the States 

96- 19 Assessment and Analysis of School-Level Expenditures 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

1 999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1 999-1 6 Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 

Finance - postsecondary 

97- 27 Pilot Test of IPEDS Finance Survey 

Finance - private schools 

95- 17 Estimates of Expenditures for Private K-12 Schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97- 22 Collection of Private School Finance Data: Development of a Questionnaire 

1999- 07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

Geography 

98- 04 Geographic Variations in Public Schools’ Costs 

Imputation 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1999 AAPOR Meetings 

Inflation 

97-43 Measuring Inflation in Public School Costs 

Institution data 

2000-0 1 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Instructional resources and practices 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

1 999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 



International comparisons 

97-1 1 International Comparisons of Inservice Professional Development 
97-16 International Education Expenditure Comparability Study: Final Report, Volume I 

97-17 International Education Expenditure Comparability Study: Final Report, Volume II, 

Quantitative Analysis of Expenditure Comparability 



Libraries 

94-07 Data Comparability and Public Policy: New Interest in Public Library Data Papers 
Presented at Meetings of the American Statistical Association 
97-25 1 996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 



Limited English Proficiency 

95-13 Assessing Students with Disabilities and Limited English Proficiency 
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No. 
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NCES contact 



Literacy of adults 

98-1 7 Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1 999-09a 1992 National Adult Literacy Survey: An Overview 

1 999-09b 1992 National Adult Literacy Survey: Sample Design 

1 999-09c 1 992 National Adult Literacy Survey: Weighting and Population Estimates 

1 999-09d 1 992 National Adult Literacy Survey: Development of the Survey Instruments 

1 999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates 

1 999-09f 1 992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 

Levels 

1 999-09g 1 992 National Adult Literacy Survey: Literacy Levels and the Response Probability 

Convention 

1 999- 1 1 Data Sources on Lifelong Learning Available from the National Center for Education 

Statistics 

2000- 05 Secondary Statistical Modeling With the National Assessment of Adult Literacy: 

Implications for the Design of the Background Questionnaire 



Sheida White 

Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 

Alex Sedlacek 

Lisa Hudson ‘ 

Sheida White 



Literacy of adults - international 

97-33 Adult Literacy: An International Perspective 



Marilyn Binkley 



Mathematics 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in Jeffrey Owings 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1 988 

1 999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test Dan Kasprzyk 

Results to Improve Item Construction 



Parental involvement in education 

96- 03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and Jeffrey Owings 

Issues 

97- 25 1 996 National Household Education Survey (NHES:96) Questionnaires: Kathryn Chandler 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
1 999-0 1 A Birth Cohort Study: Conceptual and Design Considerations and Rationale Jerry West 



Participation rates 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks Peter Stowe 

and Empirical Studies 

Postsecondary education 

1 999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education Lisa Hudson 

Statistics 



Postsecondary education - persistence and attainment 
98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field Aurora D’Amico 
Test Report 

1 999-1 5 Projected Postsecondary Outcomes of 1 992 High School Graduates Aurora D’Amico 



Postsecondary education - staff 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists Linda Zimbler 

2000-0 1 1 999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report Linda Zimbler 



Private schools 

96- 1 6 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97-22 Collection of Private School Finance Data: Development of a Questionnaire 



Stephen Brough man 
Stephen Brough man 

Stephen Broughman 



No. 
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NCES contact 



Projections of education statistics 

1 999-1 5 Projected Postsecondary Outcomes of 1 992 High School Graduates 

Public school finance 

1 999-1 6 Measuring Resources in Education: From Accounting to the Resource Cost Model 

Approach 



Public schools 

97- 43 Measuring Inflation in Public School Costs 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

98-04 Geographic Variations in Public Schools’ Costs 

1 999-02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 

Public schools - secondary 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Reform, educational 

96-03 National Education Longitudinal Study of 1 988 (NELS:88) Research Framework and 
Issues 

Response rates 

98-02 Response Variance in the 1 993-94 Schools and Staffing Survey: A Reinterview Report 

School districts, public 

98-07 Decennial Census School District Project Planning Report 

1 999-03 Evaluation of the 1 996-97 Nonfiscal Common Core of Data Surveys Data Collection, 
Processing, and Editing Cycle 

School districts, public - demographics of 

96- 04 Census Mapping Project/School District Data Book 

Schools 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

1 999- 03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

Schools - safety and discipline 

97-09 Status of Data on Crime and Violence in Schools: Final Report 

Software evaluation 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 

Variances from NCES Data Sets 

Staff 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Staff - higher education institutions 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

State 

1 999-03 Evaluation of the 1 996-97 Nonfiscal Common Core of Data Surveys Data Collection, 
Processing, and Editing Cycle 
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No. 
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NCES contact 



Statistical methodology 

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But 
Thought You Could Never Understand 

Students with disabilities 

95- 13 Assessing Students with Disabilities and Limited English Proficiency 

Survey methodology 

96- 17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 

97- 15 Customer Service Survey: Common Core of Data Coordinators 

97- 35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996 

National Household Education Survey 

98- 06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second 

Follow-Up: Final Methodology Report 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
1 999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1 999- 1 7 Secondary Use of the Schools and Staffing Survey Data 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1999 AAPOR Meetings 

Teachers 

98-13 Response Variance in the 1994-95 Teacher Follow-up Survey 

1999-14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 

Teachers - instructional practices of 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Teachers - opinions regarding safety 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Teachers - performance evaluations 

1999-04 Measuring Teacher Qualifications 

Teachers - qualifications of 

1 999- 04 Measuring Teacher Qualifications 

Teachers - salaries of 

94- 05 Cost-of-Education Differentials Across the States 

Variance estimation 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 

Variances from NCES Data Sets 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1 998 and 1 999 ASA and 
1999 AAPOR Meetings 

Violence 

97-09 Status of Data on Crime and Violence in Schools: Final Report 

Vocational education 

95- 12 Rural Education Data User’s Guide 

1 999-05 Procedures Guide for Transcript Studies 
1 999-06 1 998 Revision of the Secondary School Taxonomy 
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